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participant data 

Stephen Burgess, Ian R. White, Matthieu Resche-Rigon 
and Angela M. Wood* f 

Multiple imputation is a strategy for the analysis of incomplete data such that the impact of the missingness 
on the power and bias of estimates is mitigated. When data from multiple studies are collated, we can propose 
both within-study and multilevel imputation models to impute missing data on covariates. It is not clear how to 
choose between imputation models or how to combine imputation and inverse-variance weighted meta-analysis 
methods. This is especially important as often different studies measure data on different variables, meaning 
that we may need to impute data on a variable which is systematically missing in a particular study. In this 
paper, we consider a simulation analysis of sporadically missing data in a single covariate with a linear analysis 
model and discuss how the results would be applicable to the case of systematically missing data. We find in 
this context that ensuring the congeniality of the imputation and analysis models is important to give correct 
standard errors and confidence intervals. For example, if the analysis model allows between-study heterogeneity 
of a parameter, then we should incorporate this heterogeneity into the imputation model to maintain the con- 
geniality of the two models. In an inverse-variance weighted meta-analysis, we should impute missing data and 
apply Rubin's rules at the study level prior to meta-analysis, rather than meta-analyzing each of the multiple 
imputations and then combining the meta-analysis estimates using Rubin's rules. We illustrate the results using 
data from the Emerging Risk Factors Collaboration. © 2013 The Authors. Statistics in Medicine published by 
John Wiley & Sons Ltd. 
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1. Introduction 

Missing data are data whose values are not available. This may be for a number of different reasons both 
within and outside the control of investigators. Factors within their control tend to lead to systematic 
patterns of missing data, such as data on a variable being missing for an entire subset of the study pop- 
ulation. This may be because the measurement of a variable was only undertaken in a few studies, for 
example, due to the cost of measurement. Factors outside the control of the investigators tend to lead 
to sporadically missing data, where data on a variable are missing for a few individuals with no clear 
pattern to the missingness. 

Missing data are classified as missing completely at random (MCAR), missing at random (MAR), or 
missing not at random (MNAR) depending on whether the probability of data being missing is indepen- 
dent of the true values of the missing data (MCAR), depends only on observed data (MAR), or depends 
additionally on unobserved data (MNAR) [1, 2]. If the data are MCAR, then the missing values are dis- 
tributed identically to the measured values. If the data are MAR, then the distributions of the missing and 
measured values are the same conditional on measured covariates. If the data are MNAR, the conditional 
distribution of the missing values differs from that of the measured values. 
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There are several possible approaches with missing data. The most common approach is to ignore 
individuals with missing data entirely: a complete-case analysis. More sophisticated approaches are 
available, such as multiple imputation [3], Multiple imputation under a MAR assumption is increasingly 
being used in applied research because of recent software development. In multiple imputation, missing 
values are imputed several times by drawing random values from the conditional distribution of the miss- 
ing values according to a specified imputation model using observed data values to form a completed 
dataset. The parameter estimates and standard errors from each of these imputed datasets are combined 
using formulae known as Rubin's rules [4]. There are two main advantages to a multiple imputation 
analysis over a complete-case analysis: 

(1) Power: Observations with partial missingness may still be informative for the estimate of 
interest, especially if missingness is in a single variable. An efficient analysis should include 
all relevant information. 

(2) Bias: If the missing data are MAR, a complete-case analysis can introduce bias, whereas 
correctly specified multiple imputation estimates are unbiased [5]. 

In this paper, we consider the specific context of multiple imputation for missing data in an indi- 
vidual participant data (IPD) meta-analysis [6]. A meta-analysis is an analysis of data from multiple 
sources to give a single pooled value representing the overall estimate of the parameter of interest using 
the totality of the available data. Often, by necessity, a meta-analysis is performed on summarized data 
published by each study. In an IPD meta-analysis, the original data on the study participants is avail- 
able for analysis. This facilitates hierarchical analyses, where the analysis of multilevel data can be 
performed in a single step [7], as opposed to the common two-stage inverse- weighted meta-analysis 
method. 

The main difficulty with performing and interpreting meta-analyses is between-study variability [8]. 
This consists of both statistical heterogeneity due to differences in populations, such that coefficients can- 
not realistically be assumed to be constant across studies, and variability due to the investigators, such 
as the choice of variables measured in each study or the definition of variables. IPD enable the assess- 
ment of statistical heterogeneity and the standardization of analyses across studies [9]. Additionally, 
detailed analyses can be performed with individual-level data, which would not generally be possible 
with summarized data, such as multiple imputation [10, 1 1]. 

The imputation of missing data presents specific challenges in a meta-analysis context. For example, 
if a covariate represents an important confounder for an association, missing data methods can be used 
to impute sporadically missing data in the covariate, although it is not clear whether it would be optimal 
to impute data in each study separately or in all studies using a hierarchical model. If the covariate has 
not been measured in a study, it is unclear how to impute data on this variable using information from 
other studies. Previous work has shown that imputation of covariates across multiple studies can lead to 
inconsistencies in estimation [12]. 

The structure of this paper is as follows. We first introduce methods for the analysis of data 
from multiple studies and the imputation of missing data in this context (Section 2). Two particu- 
lar issues considered are the following: (i) the congeniality of the imputation and analysis models 
[13] and (ii) the correct order to apply the combination of imputation estimates using Rubin's rules 
and the pooling of study estimates using an inverse-variance weighted meta-analysis. We present a 
simulation study to investigate the behavior of estimates using the analysis and imputation methods 
previously introduced (Section 3). We illustrate the methods with an analysis of the association of 
low-density lipoprotein cholesterol (LDL-C) with blood pressure using data from the Emerging Risk 
Factors Collaboration (ERFC) [14] (Section 4). We conclude by discussing the findings of the paper 
and potential avenues for future work (Section 5). The methods and simulations considered in this 
paper mainly relate to sporadically missing data. Issues relating to systematically missing data are left 
for discussion. 



2. Meta-analysis and multiple imputation models 

We consider a situation where we have two continuous covariates (Xi and X2) and a continuous out- 
come (Y) in multiple studies. We present methods for the analysis of such data and for the imputation of 
missing data on one of the covariates. 
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2.1. Meta-analysis models 

The three meta-analysis methods considered here are a homoscedastic stratified analysis, an inverse- 
variance weighted fixed-effect analysis, and an inverse-variance weighted random-effects analysis using 
the DerSimonian and Laird estimate of between-study heterogeneity [15]. 

In the homoscedastic stratified analysis, we estimate the following model using linear regression: 

yis — fas + faXiis + fax 2 is + €is 



where the subscript i is used to index individual participants and s to index studies. This model 
assumes constant coefficients fa and fa in the regression model for both X\ and X 2 , with fixed study- 
specific intercepts fa s . We could perform a more sophisticated heteroscedastic stratified analysis using 
a hierarchical model to allow for heterogeneity in the variance of the error between studies (that is, 
£is ~ N (0, of)) or using random study-level effects on the regression coefficients fa and fa. We do 
not consider these analyses here for reasons of focus and brevity of presentation. 

In the inverse-variance weighted analyses, we initially fit a separate model in each study using linear 
regression: 

yis — fas + fasXlis + fas*2is + ffj 

We then combine the estimated coefficients fa s and fas using inverse-variance weighting. We can use 
either a fixed-effect or a random-effects model [16]. In this paper, we consider that an estimate either 
fa or fa is of interest; we do not consider multivariate meta-analyses for the joint distribution of the 
estimates of fa and fa [17]. 



2.2. Multiple-imputation models 

We investigate two models for imputing missing data: a stratified model and a within-study model. 

In the (homoscedastic) stratified imputation method, we impute missing data by using the following 
model with fixed study-level intercepts: 

X2is = Uos + OllXUs + C(2yis + €is 



This model uses the same ot\ and a 2 parameters in each study, fixed study-level intercept terms, and 
assumes that the error distribution is homogeneous across studies. Aside from its simplicity and avail- 
ability in a wide range of statistical software packages, an advantage of this model is the ability to 
impute data on a systematically missing covariate. In this case, we cannot estimate the study-specific 
intercept (a 0s ) from data. For the linear analysis models considered in this paper, we can fix a 0 s at an 
arbitrary value (say, zero) as the value of ao s affects only the study-specific intercept term (fa s ) in the 
analysis model. 

In the within-study imputation method, we impute missing data by using the following model: 



X2is = OCOs + OllsXliS + 0i 2s yis + €is 

This model uses different a\ s and a 2s parameters in each study and allows the residual error variance to 
differ between studies. In practice, we impute data in each study separately. 

Alternatively, we could assume a heteroscedastic stratified imputation model, where the same param- 
eters (a\ and a 2 ) are used in each study as in equation (3) but the variance of the error distribution (a 2 ) 
is different in each study. Although this may be a better model in many cases, it is unclear what value of 
the variance should be taken if no measurements have been made of the covariate in a particular study, 
although a hierarchical model for the variances may be possible [11]. We did not consider this in this 
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Figure 1. Schematic diagram illustrating the two approaches for combining Rubin's rules and inverse-variance 
weighted meta-analysis. Curved braces indicate application of Rubin's rules, square braces indicate application 

of meta-analysis. 



paper because of similarity to the other models considered; additionally, we prioritize models, which are 
available for implementation in standard software for multiple imputation, for investigation. 

2.3. Congeniality of imputation and analysis models 

An important consideration for a multiple imputation analysis is the congeniality of the models used in 
imputation and analysis of data. Imputation and analysis models are compatible if a joint model exists 
under which both models are conditionals [18]. The concept of congeniality, introduced by Meng [13] 
in the context of multiple imputation, states that as follows: (i) given complete data, the analysis model 
asymptotically gives the same mean and variance estimates as the posterior mean and variance from 
a Bayesian joint model, and (ii) given incomplete data, the imputation model gives the same posterior 
predictive distribution for missing values as the Bayesian joint model. Congeniality is similar to com- 
patibility in a non-Bayesian context with the regularity condition that the priors in the Bayesian model 
are nonzero over the entire parameter space. 

2.4. Combining Rubin's rules and inverse-variance weighted meta-analysis 

In an inverse-variance weighted meta-analysis, there are two ways of producing a single estimate from 
several multiply imputed studies: pooling across studies by meta-analysis separately for each imputed 
dataset and then combining the meta-analysis estimates using Rubin's rules, or combining estimates 
using Rubin's rules for each study and then meta-analyzing the combined estimates across studies. 
Figure 1 shows a schematic diagram of the two approaches. 

3. Simulation study 

To assess the performance of methods for imputing missing data in a particular meta-analysis con- 
text, especially with regard to the issues of congeniality and the order of applying Rubin's rules and 
meta-analysis in a inverse-variance weighted meta-analysis, we perform a simulation study. 
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3.1. Simulation of data 

We simulate data on 30 studies from the following data-generating model: 



yis — Pos + PlsXiis + filsX2is + ^is 




e is ~M{0,oj) 

In the first set of simulations, each study is of equal size with 200 participants in each study. In the sec- 
ond set of simulations, studies are of unequal size with the number of participants in each study ranging 
from 125 to 275 in steps of five individuals, such that the total number of individuals in both simula- 
tions is the same. Supporting Information* presents results from the second simulation with studies of 
unequal size. 

We consider five scenarios, representing different degrees of between-study heterogeneity: where a 
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numerical value indicates that the parameter took that value for all studies, and a normal distribution 
Af(fi, a 2 ) indicates that the values of the parameter in each study were drawn independently from a 
normal distribution. 

Scenario 1 is the most homogeneous model considered, and we add heterogeneity sequentially by 
drawing the model parameters from a normal distribution to allow for between-study variability. We 
note that the homoscedastic stratified model given in equation (1) is a correctly specified analysis model 
in Scenario 1. A heteroscedastic stratified model is a correctly specified analysis model in Scenario 2 
(and should give consistent estimates in Scenario 1). The fixed-effect method is correctly specified for 
Pi in Scenarios 3 and 4 (and should give consistent estimates for both Pi and P2 in Scenarios 1 and 2). 
We correctly specify the random-effects method for both Pi and p 2 in Scenario 5 (and should give 
consistent estimates for both parameters in each of the other scenarios). 

Similarly, the stratified imputation model is a correctly specified imputation model in Scenario 1, but 
not in any of the other scenarios. The within-study imputation model should give consistent imputed 
values in all scenarios. 

In each scenario, we create 1000 simulated datasets for analysis. We assume that Y and Xi have no 
missing observations and only consider missingness in X 2 . If itis is the probability that the observation 
Xi S is missing, we generate approximately 50% sporadically missing data in X 2 using a MAR model, 
where missingness depends on the observed value of Xi (which is correlated with X 2 ): 

7t is = expit(xi^) (6) 

where expit(x) = (1 + exp(— x)) _1 is the inverse of the logit function. We use this large missingness 
rate to illustrate the issues in parameter estimation with missing data more clearly. 

In this paper, we used five imputations for each dataset for computational reasons. In a practical 
application, more imputations would ideally be used. We generated multiply imputed datasets in stata 
(StataCorp, College Station, Texas, USA) [19]; we performed subsequent analyses of the datasets in R 
(R Foundation for Statistical Computing, Vienna, Austria)[20]. 

Tables I and II for the equal sized studies and Tables SA1 and SA2 for the unequal sized studies show 
the results of stratified, fixed-effects, and random-effects meta-analysis methods. Supporting Information 
shows alternative tables displaying the same results, but grouped by imputation method rather than sce- 
nario. In Tables I and II, we show results from scenarios where the analysis model is misspecified with a 
shaded background, whereas we show results from scenarios where the imputation model is misspecified 
in italics. 



^Supporting information may be found in the online version of this article. 



© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. 



Statist. Med. 2013, 32 4499^514 



Statistics 

in Medicine 



S. BURGESS ETAL. 



f; 


o 


on 










r- 


cn 


CS 


NO 


Os 


iri 




-t 






on 








Cs 


oc 


tt 


o 


ON 


CON 


ON 


ON 


On 


o\ 


ON 




■Os 


on 


Os 


i- 


CN 


SO 




o 




cn 


OS 


so 


T- 


OS 


ir> 




cn 














— > 








o 


o 


o 


o 


o 


o 


o 


o 






O 


O 


d 


d 


d 


d 


d 


d 


d 


d 


cs 


cs 


d 


d 


m 




so 


SO 


NO 






ON 


«n 


■» 


m 






cn 














— ' 


' — 1 






o 


o 


o 


o 


o 


o 


o 


o 


cs 


© 


o 


o 


d 


d 


d 


d 


d 


d 


d 


d 


cs 


cs 


d 


d 


c 


o 


c 


o 


oc 


r- 


c 


o 


CN 


cs 


oo 


oc 


c 


o 


c 


c 


ON 


CN 


c 


c 




o 


c> 


Os 




m 


on 


m 


cn 


cn 


on 


m 


CM 


<N 


CN 


C-s| 


d 


d 


d 


d 


d 


d 


d 


d 


cs 


CS 


d 


d 



|g>§ 

° > 2 

O ° 
on u >? 
w 73 2 

^? 

o 
8 



£ o 

H 

on h 



OQ. | 

Cl> cm 
es d 
d ra 

to £ 

o o 
S 1 

0 H 
d 0 

■Sq 

1 > 

■5 x) 

» ■§ 

w> d 

O CS 

>s M 
« "a 



3 



g 

73 



a d 
s a 
* -s 

s 3 

2 & 

Is 

3 ca 
"d 

2 1 



cS 



CS jS 



.2 u 
cs B 

It 

ft o 
u 

£ -d 
73 bp 

d d 

cs M 
^ cs 

U CJ 

co 

CS O 

o d 
3 3 
"ft s 

o 2 

^ cs 
cs d 

s3 BQ 

% 



73 



eg 

'■B 

o ^ 



ft 



. d 

>n cs 

3 " 

3 u 

«5 ^2 



r=5 .S 



o 

73 

8 

a 

Pi 



cu d 

^ "55 

§ 1 

T3 O 

& Z 

-c o 

.Sp-s 

(U ft 

S -a 

8 1 



a _ 

> -a 

Lh CD 

| s 
.9 

tu 9 

= ^ 

CL) O 

§ t 

# ? 

>o cs 

J3 C 

Mh O 

--s <d 

o g 

P. -d 



d 



■3 W 



73 

cd 

1 Q 



73 

30 



a\ so <n -h q o\ 
vt' cn cn r-H rsi ^' 

Os ON OS OS OS ON 



q so oi \o r- so 
so cn ^ Ov cn rn 
OS OS on oc OS OS 



Ol 


o 


SO 


7f 


sO 


NO 


CNl 


oc 






IT! 






rl 














— i 


' — 1 






o 


O 


c 


o 


o 


o 


O 


o 




cs 


o 


o 


rH d 


d 


d 


d 


d 


d 


r-i d 


d 




CS 


d 


d 







SO 


NC 


sO 






CN 










O 


o 


c 


o 


5 


c 


d 


d 


d 


d 


d 


d 



o 

i 
§ 

o o cs 
d d o 



o o 
d d 



c 


o 


c 


c 


oo 


£■ 


c 


c 


— < 


<~Nl 


oc 


r-~ 


c 


o 


o 


c 


ON 


ON 


o 


c 


Os. 


■Os 


ON 


On 


on 


m 


on 


m 


CN 


CI 


on 




CM 


Os) 




Cn| 


d 


d 


d 


d 


d 


d 


d 


d 


CS 


cs 


d 


d 











CN) 


00 




CNj 


sC 


7f 


s«n 




ON 


ON 


On 





rn 




so 


sO 




CN 






s 


o 


c 


s 


d 


d 


d 


d 



NtHlO 
— CN| —H 

o o cs 



o o 







so 


sO 




CN 






o 


o 


c 


o 


d 


d 


d 


d 



c 




c 


r- 


c 


o 


c 


ON 


m 


cn 


m 


CN 


d 


d 


d 


d 



on 




so 






C! 






o 


o 


cs 


1 


d 


d 


Cs 






o 




Cs 




c 


o 


cs 


ON 


on 




<sn 




d 


d 


Cs 


S 



a, 
3 



< 



s 

c 

u 



6 E 



5 



s s 



s e 



U U co 



J5 S 



© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. 



Statist. Med. 2013, 32 4499^1514 



S. BURGESS ETAL. 



Statistics 

in Medicine 



in 


oc 


cs 


Oo 




r~ 


OS 


ON 


■ — i 


rs. 




CS 








CS 




cn 


in 




K 




oc 




in 




K 




CO 




in 






cn 


NO 




on 


on 


C\ 


On 


on 


ON 


ON 


ON 


'On 


On. 


ON 


ON 


ON. 


ON 


On 


'On 


ON 


ON 


cn 


C 


Cs 


K 


On 


NO 


cn 


o 






ON 


NO 


cn 


r- 


NC 




f- 


■* 




r-l 


ri 


— i 








cs 


cs 








(-Nl 


rs 


i 


CS 


cs 


CN 


o 


C 


Cs 




o 


o 


5 


O 


oo 


cs 


o 


c 


c 


o 


cs 


Cs 


o 


O 


d 


C 


ci 


Oo' 


d 


d 


d 


d 


ci 




d 


d 


d 


d 


ci 


CS 


d 


d 


CS 


ON 


no 


NO 


in 


m 


rs 


ON 






m 


m 


cn 


r- 


> 




■* 


■* 








1 — i 










' — i 


— 1 






<N 


CS 






n 


ri 


o 


o 


cs 




o 


o 


o 


o 


Cs 


cs 


o 


o 


o 


O 




§ 


O 


O 


d 


d 


ci 


ci 


d 


d 


d 


d 


ci 


Oo 


d 


d 


d 


d 


cs 


CS 


d 


d 



o 


ON 


cn 




oc 


t- 


o 


o 


in 




ON 


oo 


on. 


On 


>n 


in 


r- 


r» 


o 


On 


On 


On 


ON 


On 


o 


o 


'On 


On. 


ON 


On 


ON 


On 


On 


'On 


ON 


ON 


CO 


("Nl 


CM 


CN 




CS 


cn 


cn 


CS 


CS 


cn! 


("Nl 


Cn) 


CS 


CN 


CM 


CM 


ci 


d 


d 


ci 


Cs 


d 


d 


d 


d 


ci 


ci 


d 


d 


d 


d 


o* 


Ci 


d 


d 





NC 


00 


On 






in 


CO 




On. 


o 




■* 




K 






cn 


-f 


cn 


'OC 


NO 




cn 


ON 


ON 


oc 


OC' 


ON. 


On 


ON. 


ON 


OC 


oo 


ON 


on 



in 


t-; 


O 


00 


oo 


CS 


in 




55 


00 


cs 




no 


r- 




NO 


r~ 





(N oo ^ > in in 

T-H ^ -~ 



N » > W \t 



O 

d 



o o 
d d 



o 

•s 

s 

p 

r « oi o x <c 'f. if , 

^ J ^ i — ( 



o o 
^ d d 
o 



o o 
d d 



cn oo >n > 
o o o o 
dodo 



cs 


ON 


NO 


K 


m 


m 














o 


o 


o 




o 


o 


d 


d 


ci 


Ci 


d 


d 



cs 


ON 






m 


m 














o 


o 


Oo 


O 


o 


o 


d 


d 


Co 


ci 


d 


d 



in 


on 


NO 


NO 


NO 


cs 


cs 


r-i 




CS 


o 


o 


Cs 


Cs 


O 


d 


d 


Ci 


Ci 


d 



o 


ON 




' 1 


oc 


t- 


o 


o 


cn 


N\f 


ON 


oc 


o 


On 


On 


O- 


ON 


On 


o 


o 


'On 


On. 


ON 


ON 


cn 


("Nl 


CM 


CN 


cs 


rs 




cn 




CS 


cs 


("Nl 


d 


d 


ci 


Oo 


d 


d 


d 


d 


Oo 


ci 


d 


d 



ON 


ON 


cs 




00 


00 


ON 


ON 


'On 


'On 


ON 


ON 


cs 


cs 


CS 


cs 


cs 


cs 


d 


d 


ci 


ci 


d 


d 





On 


00 




C-; 


ON 


cn 


in 






in d 


cn 




cs 


ON 


On 


On 


ON On 


ON 


on 


ON 



o 


NO 


on 




NO 


in 


?; 


ON 


r~ 


CO 











NO 


NO 










o 


c 


o 


o 


d 


d 


ci 


d 



Nt N K 
^ M N 
O O CS 

odd 



o 
d 



2 


cs 








cs 


^< 




c 


o 


Cs 


o 


d 


d 


Ci 


d 



-* 




NO 


NO 




cs 






o 


o 


Cs 


o 


d 


d 


ci 


d 



NO 


cn 


00 


00 


ON 


>n 


in 




IN 


1 — i 


^ cs 


CS 


CS 


cs 


o 


o 


CS 


o o 


o 


Cs 


o 


d 


d 


Cs 


d d 


d 


Ci 


d 



o 


On 


Cs 


r- t-i 






00 


o 


ON 


Cs 


ON O 


o 


Cs 


ON 


cn 


cs 


cn 


cs cn 


cn 


cn 


cs 


d 


d 


ci 


d d 


d 


Cs 


d 



On 


ON 


on 


NO 


On 


ON 


On 


ON 


cs 


cs 


CM 


cs 


d 


d 


Ci 


d 



-s 

< 



g 



Ci 

c< 



I t 



3 3 



< 

s 



[si 

Ci 



< 

S 



6 6 -a 



g, h a .a 



U U oo as 



2 3 

i i 



C 

s 

c 

c 

I 



© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. 



Statist. Med. 2013, 32 4499^1514 



Statistics 

in Medicine 



S. BURGESS ETAL. 



oo 


-H 


ON 




oc 


in 


in 






in 






in 


in 


co 


o 


r-- 


co 


in 




© 




oc 




o 


ON 


ON 


ON 


ON 


ON 


ON 


ON 


On. 




ON 


On 


"* 


o 


oc 


in 


CO 


OC 


CN 


oc 




■On 


CN 


NO 




r] 






rj 








CN 


' — i 


CN 




o 


O 


o 


o 


o 


5 


c 


o 


o 


© 


c 


o 


d 


d 


d 


d 


d 


d 


d 


d 


©' 


©' 


d 


d 


en 


o 


oo 


oc 


oc 


on. 


CN 


r- 


On. 


© 




NO 


















— 1 


<• 1 






o 


O 


o 


o 


o 


o 


c 


o 


© 


© 


o 


o 


d 


d 


d 


d 


d 


d 


d 


d 


©' 


©' 


d 


d 


o 


o 


o 


ON 


in 


ON 


o 


o 


— 1 




IT, 


OO 


o 


o 


o 


ON 


On 


ON 


o 


o 


oo 


ts 


ON 


ON 


NO 


no 


NO 


in 


in 


in 


NO 


NO 


in 


in 


in 


in 


o 


o 


o 


o 


o 


o 


o 


o 


© 


© 


o 


o 


q 


CO 


cN 


co 


CN 


c 


co 


oc 


© 


in 


oc 


q 


in 


CO 


co 


o 


CN 






CO 


©' 


oc 


CN 


cn 


on 


ON 


ON 


ON 


0 s . 


0 s . 


ON 


ON 




CV] 


ON 


On 


co 


On 




in 


r- 


i- 


CN 


NO 


rv 


in 


in 


in 


















— i 


■ — i 






o 


O 


c 


o 


o 


c 


o 


s 




© 


s 


o 


d 


d 


d 


d 


d 


d cni 


d 


d 


©' 


©' 


d 


d 












o 








































































s 














m 


o 


oc 


oc 


oo 


ON 8 


CN 




On. 


'On 


NC 


NO 




ri 














— 1 


' — i 






o 


o 


o 


o 


o 


S <« 


c 


o 


© 


© 


o 


o 


d 


d 


d 


d 


d 


d 


d 


d 


©' 


©' 


d 


d 


o 


o 


o 


ON 


t- 


o 


o 


o 


CN 


NC 




ON 


o 


o 


o 


ON 


ON 


o 


o 


o 


NO 


NC 


ON 


ON 


NO 


NO 


NO 


in 


in 


NO 


NO 


NO 


l/-j 


l/~. 


in 


in 


d 


d 


d 


d 


d 


d 


d 


d 
1 


©' 


©' 


d 


d 


q 


On 












CN 


CM 




ro 




in 


co 


CO 




in 




in 








NO 




on 


On 


ON 




oc 




ON 


ON 


On 




00 




























co 


On 


t- 




oc 






o 


00 




ON 


















cn 


— i 








o 


O 


c 




o 




c 


o 


o 




s 




d 


O 


d 




d 




d 


d 


o' 




d 





lis 

u > 2 
o o 'j* 
cn u ^ 

w T3 2 

d ^ ■§ 
3 



•I E 



,3 E 

o U 

E "g 

.2 Q 
« & 

5 a 

ft o 

E 1 
9 '> 

B -S 
3 "2 

6 -a 

m 3 



o u 
CS E 

3 >N 

ft .tj 
u 

3 



CD 
3 
3, 

TO 



* 3 
s 3 

2 S 

Is 

3 M 

a 

t & 

-3 t» 



T3 

CD 



TO M 

a .5 



C3 



f s 

g «: 
o o .2 



C>3 CD 
I 1/3 

B o 

CD > 
ft* 

s .a 

O o, 
§»■§ 

S3 E 
ft i/> 
£ 
o 

TO 
3 



M 3 

9! m 

A o 

M 'cm 



2 W 

S ^3 



o 
— 

3 

3 

pi 



,2 td-s M) 
ft o 

3 cj 
2 o 



CD 3 

§ 1 

T3 O 

-3 O 

.Sp-s 

CD ft 

* -a 

8 1 



C3 CD 
> ft 



. C3 

> T3 

Lh CD 

3 ft 

CD E 

*'i 

CL) O 

3 ' s 

§ t 

# | 

in co 

CD 3 

CtH O 

O CD 

P. -a. 



>> w 

a 



•a Q 



T3 

OS 



O O O 

odd 



o o o 
o o o 

NO NO NO 

odd 

I I I 



o 
d 



o 



n* O ON 
' — 1 CN 1 — i 

o o © 
do© 



o 
d 



o 


o 


© in 


o 


o 


© 00 


NO 


NO 


no in 


d 
1 


d 
1 


©' d 



a, 
3 



< 
2 



S 

c 
U 



6 E 



s s 



S E 



nil 

CO > ^ 



© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. 



Statist. Med. 2013, 32 4499^1514 



S. BURGESS ETAL. 



Statistics 

in Medicine 



oc 


CS 






NO 


m 


oc 








m 


no 




m 


IS. 




cn 


NO 


in 


in 


in 




oo 


in 


-r 




NO 


' 


in 


en 


en 


-T 




On 


NO 




On 


ON 


00 


no 


ON 


ON 


ON 


ON 


30 


00 


On 


ON 


ON 


ON 


oo 


N. 


ON 


ON 
























Q 












Q 


s 


o 




3 


o 


o 


cn 
c 


3 


3 


1' 


3 


3 


rn 
O 


3 


3 


3 


3 


3 


d 


d 




©' 


d 


d 


d 


d 






d 


d 


d 


d 


©' 


© 


c 


d 




r- 


© 


© 


r- 


NO 


c 




"-i 








oo 




'On 


© 


o 


o 


s 


o 


8 




o 


o 


3 


3 


3 


s 


3 


i 


m 
o 


3 


3 


3 


3 


3 


C 


o 




© 


d 


d 


c 


d 


© 


©' 


d 


d 


d 


d 


©' 


© 


c 


d 



U 



2 W 

S t*3 



0 Q 



its 

•a Q 

U 



'« Q 

in 



T3 
OS 



O Ol »i o 

O ON K N, 

no in in 
do©©' 

I I I I 



in oo 

ON On 

in in 

d d 

I I 



(N Cn) C\l On no on 

O O O0 t\ ON ON 

no no in in in in 

d d ci ci d d 

I I I I I I 



oc on k in m no 

on on N, K on on 

in in in in in in 

d d o o d d 

I I I I I I 



p K on — i rJ 
in \t K cd \t ri 

ON ON Cn) Cn) on ON 



cnI r-~ t\ in m in 

^-t ^ . 1 



q 


in 


On 


00 




NO -tf 


o 


rNi 


On 




en 


oc 




On 


NO 


ON 


on 


d 


K 


<N 


On 


in 


m 


m 


CM 


tN 




^- m 


m 


rNi 









o o 
d d 



© © 
©' ©' 



o o 
d d 



m r- on 

o o 
d d 



© NO NO 
. Cn) -h — 

© © o o 
©'©'do 



CNl 
O 


o 


00 

o 


in 
© 


NO 

o 


m 
o 




in r- 
o o 


00 

© 


•n 
"-^ 

o 


NO 

o 


m 

s 


d 


d 


o 


©' 


d 


d 


m 


d d 


©' 


©' 


d 


d 














o 




































m 
o 


o 


in 

3 


in 

s 


NO 

-r 

o 


o 


c 

u 
o 
in 


in oo 
-tf n* 
o o 


<n 

3 


^n 

s 


NO 
TI- 
CS 


-t 

o 


d 


d 


© 


©" 


d 


d 




d d 


©' 


©' 


d 


d 



O ^ CQ h ^ 

O ^ c> a\ 

\o ir> iri in in io 



o o 



o o 



C-1 


<n 


<n 


in 


ON 


00 


o 


o 


"1- 




On 


o 


nO 


NO 


>n 


5; 


in 


NO 


d 

1 


d 
1 


o' 

1 


1 


d 
1 


d 
1 



C\ O O l> ^ 

a o > ^- a\ o 

ifl nO ^ I/) if] \D 

d d ci ci 
I I 



o o 



NO 


in 




— 1 CN) 






in — < 




rn 


NO 








in d 






d en 


NO* 


rn 


On' 


ON 


ON 


on 


oo »n 


NO 


NO 


no in 


NO 


NO 


m 





o 


00 


00 




r i 


1 — i 




o 


o 


© 


o 


d 


d 


©' 


d 



2 


o 


ON 


On 




rsi 






o 


c 


© 


O 


d 


d 


©' 


d 



2 


o 


On 


On 




<-i 






3 


o 


© 


O 


d 


d 


©' 


d 





o 


on 


ON O 


CNl 


OnI 


O ON 




© 


o 


3 


o 


© 


3 3 


3 


3 


o o 


3 


3 


3 


d 


d 


©' 


d d 


d 


©' 


d d 


d 


©■ 


d 



On 


On 


© 


in 


ON 


On. 


© 


00 


m 


in 


NO 


m 


d 
1 


d 
1 


© 
1 


d 
1 



rNi 


m 


<N1 


r- 


o 


o 


© 


00 


NO 


NO 


NO 


in 


d 
1 


d 
1 


© 
1 


d 
1 



00 


ON 


© 




ON 


ON 


© 


00 


m 


in 


NO 


m 


d 
1 


d 
1 


© 

1 


d 

1 



g 



< 

3 



< 
S 



a; < 



< 



■data 


■case 


imp (] 


imp (i 










")mple 


)mple 


; — 


'1- 


rati 1 


ratii 


U 


U 







e e 



e e 



a, 
3 



< 



S E 



a a 



U U w w 



J5 S 



= 

c 
U 



I 



3 ^ 



© 2013 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. 



Statist. Med. 2013, 32 4499^1514 



Statistics 

in Medicine 



S. BURGESS ETAL. 



3.2. Results of complete -data and complete-case analyses 

Before considering the imputation of missing data, we discuss complete-data and complete-case analy- 
ses estimates of both fi\ (Table I) and (Table II). In the complete-data analysis, we analyzed data from 
all individuals prior to the introduction of missing values. In the complete-case analysis, we excluded 
individuals with missing data values from analysis. We present results for each of the three meta-analysis 
methods: the mean estimate across simulations, the standard deviation of estimates, the mean of the esti- 
mated standard errors, and the coverage of the 95% confidence interval for the true parameter of interest 
(or the mean of the parameter's distribution when there is heterogeneity in the parameter of interest). 
In an ideal analysis, the mean estimate across studies should be close to the true parameter value, the 
empirical standard deviation of the estimates should be close to the mean standard error estimate, and 
the coverage should be close to 95%. We constructed confidence intervals assuming normal distributions 
of the parameters of interest. 

The pooled estimate from each of the methods shows little bias throughout even when the model is 
misspecified. The stratified and the fixed-effect analyses give good estimates when the parameter of inter- 
est (i.e., Pi or /3 2 ) is fixed between studies, with some reduction in coverage and less efficient estimates 
for the stratified method when there is some between-study variability in other parameters. However, 
both the stratified and the fixed-effect methods underestimate variance when the parameter of interest is 
heterogeneous. The random-effects meta-analysis method gives marginally larger standard errors than 
the fixed-effect method when there is no true heterogeneity in the parameter of interest, but gives much 
better coverage when heterogeneity is present. Coverage in the random-effects meta-analysis is known 
to be theoretically underestimated because of uncertainty in the heterogeneity not being acknowledged 
[21]; this does not seem to be a serious issue here as with 30 studies the heterogeneity is well estimated. 
We note that the loss of information in the complete-case analyses over the complete-data analyses is 
much less in Scenarios 4 and 5 where there is heterogeneity in the parameter of interest, than it is in the 
other scenarios. 

As seen in these simulations, the complete-case analyses are less efficient than the complete-data anal- 
yses. This motivates us to consider methods for the imputation of missing data. We note that analyses, 
which perform badly in terms of bias or coverage with complete data, are not going to perform well 
using multiple imputation methods; we should not interpret this as a failure of the multiple imputation 
method, and we should see correct specification of the analysis model as a first step before choosing 
between imputation models. 



3.3. Results of combining Rubin 's rules and inverse-variance weighted meta-analysis 

To assess the impact of the order of combining Rubin's rules and an inverse-variance weighted meta- 
analysis, we initially consider estimates with a fixed-effects and a random-effects meta-analysis model 
using the within-study imputation method, as in this case, the imputation model is correctly specified in 
all scenarios. 

Within-study imputation, fixed-effect analysis: With the fixed-effect analyses (ignoring scenarios 
where a fixed-effect analysis is not appropriate), the coverage is further below the nominal 95% that 
would be expected by chance when estimates are combined using Rubin's rules and meta-analysis 
whichever order the processes are undertaken. (The Monte Carlo standard error for the coverage, rep- 
resenting the uncertainty in the simulated results due to the limited number of simulations, is 0.7%.) 
Additionally, there is a slight but consistent bias toward the null. 

The reason that the fixed-effect analyses are undercovered and demonstrate bias is that the imputa- 
tion of missing data introduces heterogeneity into the estimates of the parameter of interest (say f}\) 
even when there was no heterogeneity in this parameter in the data-generating model for the studies. 
Even though the parameter in the data-generating model was the same in all studies, the estimates 
of the related parameter used in the imputation model for generating imputed data will be differ- 
ent, and so, a fixed-effect analysis model will be misspecified. A study with by chance a larger than 
average estimate of /?i in the available data will use this inflated estimate of fix (via the related 
parameter a 2 m the imputation model) to impute the missing data. Hence, the parameter estimate 
from a study with by chance a larger than average estimate of fi\ will be less precise than from a 
study with a smaller than average estimate. Pooling the estimates of association results in increased 
weights for studies with smaller than average estimates of f>\, and a slight downward bias in the com- 
bined estimate. This introduction of heterogeneity also leads to slight under-coverage when Rubin's 
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rules are applied before meta-analysis, as the fixed-effect assumption will no longer be valid. The bias 
and reduction in coverage levels seem to be of similar magnitude when the studies are of equal and 
unequal size. 

Within- study imputation, random-effects analysis: With the random-effects analyses, the coverage of 
the 95% confidence interval when the results are meta-analyzed then Rubin's rules are applied is con- 
servative at 97.7% or greater when there is no heterogeneity in the parameter of interest, with mean 
standard error consistently larger than the standard deviance of the estimates. When the results are com- 
bined using Rubin's rules then meta-analyzed, the coverage is close to the nominal 95% when there is 
no effect heterogeneity. When there is heterogeneity in the parameter of interest, random-effects meta- 
analysis is known to give slightly over-narrow confidence intervals as stated in Section 3.2. However, 
this is an issue with the meta-analysis method, not with the imputation method, and the coverage is close 
to that achieved in the complete-data analysis. Additionally, there is a slight but consistent bias towards 
the null. 

The reason for the overly conservative coverage is that when the multiple imputations are made, addi- 
tional heterogeneity is introduced into the imputed datasets. If the imputed datasets are meta-analyzed 
before combining by Rubin's rules, then the heterogeneity of the meta-analysis results represents the sum 
of the true heterogeneity between the studies and the heterogeneity introduced due to the imputation pro- 
cess. If the imputed datasets are combined for each study using Rubin's rules before the meta-analysis, 
then each study estimate after combining reflects the true uncertainty of the estimate using all the data 
in the study. 

Stratified imputation: If we consider the stratified imputation model in Scenario 1, the only scenario in 
which this imputation model is correctly specified, then a congenial analysis requires the meta-analysis 
to be performed before the application of Rubin's rules. This is because missing data in each study is 
imputed conditional on data in other studies, inducing a dependence between the imputed data values 
in different studies, which is not accounted for when Rubin's rules are applied at the study level. The 
inverse-variance weighted analysis models assume that estimates of the parameter of interest from each 
study are independent. In this case, the fixed-effect analysis still has slightly low coverage, whereas the 
random-effects analysis has correct coverage levels. 



3.4. Results of comparison of imputation models 

To assess the impact of different imputation methods on estimates, we compare the performance of 
estimates from each of the analysis methods. 

In general, the efficiency of the multiple imputation analyses for jf3 1 is greater than that of the 
complete-case and slightly below than that of the complete-data analyses. The efficiency for fa is similar 
to that of the complete-case analyses, with some slight improvement especially when there is hetero- 
geneity in the parameter. We see that the results obtained are most sensitive to the choice of imputation 
method. 

Stratified imputation: The results using the stratified imputation method for the inverse-variance 
weighted analyses show bias in all scenarios except Scenario 1, where there is little heterogeneity 
between the studies. This is especially marked for estimates of fa. The coverage is underestimated, 
with the mean standard error being generally less than the standard deviation of the estimates, even in 
Scenario 1 . This is because the imputation induces a correlation between data values in different stud- 
ies, which is not acknowledged in an inverse-variance weighted analysis model. Although the stratified 
analysis method is misspecified in all scenarios except Scenario 1, the results for this method are not so 
bad, with minimal bias. This may reflect the congeniality of the imputation and analysis models. The 
inverse-variance weighted meta-analysis models do not correspond to the imputation model, and so are 
uncongenial. It seems that the heteroscedasticity introduced from Scenario 2 onwards, which makes the 
imputation model misspecified, is the key feature of the generating model, which introduces bias into 
the inverse-variance weighted results. 

Within-study imputation: Using the within-study imputation method, the stratified analysis method 
gives biased estimates. The method does not perform as badly in Table I as the inverse- variance weighted 
methods with the stratified imputation method, although bias is more considerable in Table II. We 
described the behavior of estimates from inverse-variance weighted methods with the within-study impu- 
tation method in Section 3.3. In Scenario 1, using the within-study imputation and the stratified analysis 
models, the finding that there is bias and that the coverage is low runs contrary to the conventional advice 
in multiple imputation that the imputation model can be more detailed than the analysis model [22]. In 
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this scenario, the models should both lead to consistent estimates, and the imputation model is larger 
than the analysis model. However, there is bias, and the coverage is lower than the nominal 95% level. 

We conclude that the stratified imputation method should be avoided when there is heterogeneity 
between studies. Koopman et al. and Andridge made a similar finding in an applied study with a 
logistic analysis model [12] and in the context of a cluster randomized trial [23], respectively. This 
is unfortunate, as the stratified imputation model provides a method for imputing data on a covariate, 
which is completely missing in a particular study [24]. In the complete absence of data on a covari- 
ate, we must make strong assumptions not only about the relation between the covariate and other 
variables in the model but also about the error distribution of the covariate. We return to this issue in 
the discussion. 



3.5. Results with different numbers of studies 

In response to a referee's concern that the results of the simulation study may be different when fewer 
studies are including in a meta-analysis, we additionally performed the simulation study with the same 
parameters, except with five studies and with 10 studies. Tables SA3-SA6 shows the results. Similar con- 
clusions are reached in this case: Stratified analysis models are undercovered when there is heterogeneity 
between studies, even when the parameter of interest is fixed; stratified imputation models for inverse- 
variance weighted analysis models result in bias and poor coverage properties; within-study imputation 
models for a fixed-effect analysis model result in poor coverage; within-study imputation models for 
a random-effects analysis model result in poor coverage when the meta-analysis is performed before 
combining imputation estimates using Rubin's rules (MA then RR), but coverage is similar to that of 
the complete-data analysis when the study-specific imputation estimates are combined using Rubin's 
rules prior to meta-analysis (RR then MA). We underestimate the main difference from the analysis with 
30 studies that the coverage of the complete-data analyses using a random-effects analysis model (and 
therefore the coverage of a correctly specified congenial multiple imputation analysis) when there is het- 
erogeneity in the parameter of interest. As previously stated, this is a known feature of random-effects 
meta-analysis when there are few studies and the between-study heterogeneity is poorly estimated. 
This can be mitigated by using a t -distribution rather than a normal distribution to form confidence 
intervals [21]. 



4. Example: the association of low-density lipoprotein cholesterol with 
blood pressure 

We illustrate our findings with data from the ERFC on 53,723 participants from 10 studies for the 
association of LDL-C (units mmol/L) with systolic blood pressure (units mmHg) using body mass 
index (units kg/m 2 ) as a covariate. Subjects that were taken are those with complete data on sys- 
tolic blood pressure and body mass index in studies with data on LDL-C. We introduced miss- 
ing data on LDL-C for 20% of participants in each study by discarding observations completely at 
random. We report complete-data, complete-case, and imputation analyses, by using stratified and 
within-study imputation methods. We used stratified, fixed-effects, and random-effects meta-analysis 
methods. In the inverse-variance weighted analyses, results are given where Rubin's rules and meta- 
analysis have been applied in both orders. For each imputation model, we generated 50 imputed 
datasets. 

Table III shows the results. There was considerable heterogeneity between the studies, with I 2 — 77% 
(95% CI: 58%, 87%) in the complete-data meta-analysis. The only analysis where the imputation brings 
the point estimate closer to the complete-data estimate and reduces the standard error of the estimate 
(but not to be lower than that from the complete-data analysis) is the random-effects meta-analysis using 
the within-study imputation model and using Rubin's rules then meta-analyzing. This was the preferred 
method from the simulation study when there is between-study heterogeneity, where the efficiency of 
the multiple imputation analysis is close to that of the complete-data analysis. Concerningly, using the 
stratified imputation model and the random-effects analysis model, the precision of the multiple impu- 
tation analysis is greater than that of the complete-data analysis, and using the stratified analysis model, 
precision of the multiple imputation analyses is less than that of the complete-case analysis. Table SA7 
shows further details of the data in this example. 
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Table III. Regression coefficients for the association of low-density lipoprotein cholesterol (mmol/L) with 
systolic blood pressure (mmHg) adjusting for body mass index (kg/m 2 ) from complete-data, complete- 
case, and multiple imputation analyses with stratified and within-study imputation models using stratified, 
fixed-effects, and random-effects meta-analysis models: estimates with standard error (in brackets). In inverse- 
variance weighted analyses, it is indicated whether Rubin's rules were applied within each study prior to 
meta-analysis (RR then MA) or meta-analysis of imputed datasets was performed prior to combining estimates 
using Rubin's rules (MA then RR). 


Analysis model: 


Stratified 


Fixed-effect 


Random-effects 


Complete-data 
Complete-case 
Stratified imputation (MA then RR) 

(RR then MA) 

Within-study imputation (MA then RR) 

(RR then MA) 


1.219(0.078) 
1.230 (0.088) 
1.248 (0.093) 

1.236 (0.089) 


1.084 (0.069) 
1.105 (0.078) 
1.093 (0.081) 
1.099 (0.077) 
1.112(0.079) 
1.110(0.078) 


1.189 (0.225) 
1.166 (0.231) 
1.278 (0.220) 
1.278 (0.211) 
1.165 (0.239) 
1.177 (0.226) 



5. Discussion 

In this paper, we have considered combining multiple imputation and meta-analysis using simulated 
and real data. Two main issues have been addressed: the order for applying Rubin's rules and an 
inverse-variance weighted meta-analysis, and the congeniality of the imputation and analysis models. 

In our simulation study, imputing missing data from a model that allows for between-study hetero- 
geneity induced heterogeneity between studies in a meta-analysis even where there was no heterogeneity 
in the original data. This resulted in poor coverage properties in a fixed-effect meta-analysis model 
whichever order of Rubin's rules and the meta-analysis of studies was applied, even when there was no 
heterogeneity in the data-generating mechanism for the parameter of interest. A random-effects meta- 
analysis of the study-specific estimates combined by Rubin's rules (Rubin's rules then meta-analysis) 
gave pooled estimates with the correct coverage level; we overestimated confidence intervals when 
Rubin's rules were applied to pooled estimates from imputed datasets (meta-analysis then Rubin's rules). 



5.7. Congeniality of the imputation and analysis models 

Use of congenial imputation and analysis models has a more fundamental impact on meta-analysis 
results. We considered a stratified imputation model, where the same coefficients for each covariate and 
the same error distribution were assumed across studies, and a within-study imputation model, where 
the coefficients and error distributions were estimated separately for each study. The stratified imputa- 
tion model is congenial to the stratified analysis model, as both models can be derived from the same 
underlying joint model. Similarly, the within-study imputation model (with Rubin's rules applied at 
the study level) is congenial to both the fixed-effects and the random-effects inverse-variance weighted 
analysis models. Otherwise, the imputation and analysis models are not congenial. For example, for a 
stratified imputation model with an inverse- variance weighted analysis model, the imputer assumes more 
than the analyst. In this case, standard errors for the parameter of interest will generally be too small, 
and coverage may be below the nominal level. 

In our simulations, the stratified imputation model performed moderately well for the stratified analy- 
sis method, but poorly for the inverse-variance weighted analysis methods. The within-study imputation 
method performed well for the inverse-variance weighted analysis methods, especially under a random- 
effects model, but showed some bias and reduced coverage with the stratified analysis method, although 
the bias was less than for the inverse-variance weighted methods with the stratified imputation model. 
Although the stratified analysis method performs well in this paper in the absence of heterogeneity in 
the parameter of interest, we caution over its use in practice, as it is likely that additional between-study 
variability beyond that considered in this paper may be present. 

In their paper, Robins and Wang [25] gave examples showing that, when the imputer assumes more 
than the analyst: (i) if the imputation and/or analysis model are misspecified, bias in the standard error 
can be in either direction; and (ii) if the imputation and analysis models are correctly specified, bias 
in the standard error is upward. In contrast to this, we found a downward bias in the standard error in 
the case of Scenario 1 using a stratified imputation model and a fixed-effect analysis model where the 
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imputation and analysis models are correctly specified, but the imputation model additionally (correctly) 
assumes homogeneity of variance parameters, although the standard error in the random-effects analysis 
was seemingly unbiased. This appears to be a specific property of the meta-analysis context: Imputa- 
tion across studies using a stratified model induced correlation between the imputed data values, and 
between the parameter estimates from each study, leading to overly precise estimates in the fixed-effect 
analysis model. 

5.2. Imputation of systematically missing covariates 

Two methods have been proposed for the imputation of systematically missing covariates, that is, covari- 
ates, which are completely missing in a study. The first method [24] relies on imputation with a fixed 
error distribution for all studies similar to the stratified imputation model considered here. This method 
suffers from bias, which may be due to the uncongeniality of the imputation and analysis models as 
demonstrated in this paper. However, this method is for a multilevel (one-stage) analysis model, which 
is not considered in this paper, which focuses on inverse-variance weighted (two-stage) meta-analyses. 
The second method [26] relies on a multivariate meta-analysis of the regression coefficients for the vari- 
able of interest under different models of covariate adjustment. In a simple case where studies measure a 
set of covariates either U\ or (U\, U2) and a fully adjusted estimate is required, we take inference from 
the bivariate meta-analysis of regression coefficients for the model of outcome on the variable adjust- 
ing for Ui and coefficients for the model adjusting for (U\, U2). A disadvantage of this method is the 
additional complexity if studies measure multiple different combinations of covariates, and the lack of 
generalizability to missing data on the outcome or variable of interest. 

One possibility for imputing data on a completely missing covariate without making restrictive 
assumptions about its variance is by using information on measured covariates in that study to estimate 
the variance of the systematically missing covariate. We could consider a multivariate meta-analysis 
model for the standard deviation of all covariates. This would mean that studies with large variances of 
the measured covariates would have large estimated values for the variance of the missing covariate. We 
could implement this in a Bayesian framework, and we would acknowledge uncertainty in the estimate 
of the variance of the covariate throughout the model. We would require further detailed simulations to 
establish the validity and efficiency of such a method compared with established methods. 

5.3. Fixed-effects and random-effects on covariates 

The two imputation models considered in this paper can be thought of as extreme cases with respect to 
the pooling of coefficients and error distributions across studies. We could use a homoscedastic model 
for the imputation (or analysis) model, with different coefficients in each study but the same error dis- 
tribution across studies. We can perform this by allowing random-effects distributions on the regression 
coefficients in a hierarchical model [27]. We could implement such a model in a likelihood framework 
using MLwiN [28] or a Bayesian framework [29]. Theoretically, we could use a random-effects distri- 
bution for the variance in each study, although it is not clear what distribution would be appropriate 
(a normal distribution on the log-transformed standard deviation parameters could be used), or how to 
implement such a method in practice. 



5.4. Survival data 

Although the context of this paper has been for the imputation of continuous covariates with a continu- 
ous outcome, the main outcome of interest in the ERFC dataset is a survival outcome. We would need 
further simulations to establish the behavior of the methods considered for survival data. 



5.5. Conclusion 

In conclusion, with sporadically missing data in a meta-analysis, congeniality of the imputation and 
analysis models is important for obtaining valid estimates. In a random-effects meta-analysis, we should 
apply Rubin's rules at the study level prior to the pooling of study- specific estimates, and we should use 
a within-study imputation model. The scope of these conclusions is limited by the simulation nature of 
the analysis and the limitation to considering a linear analysis model with only two covariates, but we 
have no reason to believe that the findings would not also apply more generally. We need further work to 
address the issue of how to achieve congeniality of imputation and analysis models with systematically 
missing covariates. 
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